Electronic layout generation based on visual context

ABSTRACT

An electronically implemented method is disclosed for generating a layout for a composite image. The method starts with the steps of receiving ( 210 ) at least one content element to be included within the composite image and an external environment context image ( 220 ) indicative of the intended external environment of the composite image. The external environment context image is captured by an image capture device. Data is also received indicating the size of the composite image and/or the location of the composite image within the external environment context image. Based on the received data layout styles are generated ( 250 ) for the at least one content element based on features of the at least one content element, features of the external environment context image and the provided location of the composite image. Finally, based on the generated layout styles, the composite image ( 260 ) is generated comprising the at least one content element.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application claims the right of priority under 35 U.S.C. §119 basedon Australian Patent Application No. 2008202672, filed 17 Jun. 2008,which is incorporated by reference herein in its entirety as if fullyset forth herein.

TECHNICAL FIELD

The present invention relates to image layout generation and, inparticular, to methods and systems for generating layouts for compositeimages based on the visual context of the layout.

BACKGROUND

The traditional graphic design method involves three steps: explorationof the design concepts, review of the initial ideas, and refinement ofthe selected concept to make the final design product. While theexploration and the review processes rely heavily on graphic designers,the refinement method can be automated using computer systems.

In the exploration stage, the graphic designer needs to understand thepurpose of the design (e.g., what is the design for) and the context ofthe design (e.g., where and how is the design going to be displayed).The designer further needs to collect the content of the design,including text and images, and generate multiple design concepts, usingsketches, in order to create a design or a design product. The finaldesign product is usually a composite image having a particular pagelayout. A graphic designer produces one or more initial page layoutconcepts for the content using established design principles and pastexperience. The initial page layouts are then evaluated by other graphicdesigners and/or customers. Finally, a single page layout is selectedfrom the initial page layouts. The selected layout is refined byarranging and applying styles to the text and image content to producethe final composite image, referred to as a final page layout or layout.

There are existing computer aided design systems arranged for generatinga professional looking page layout in the refinement stage. Many ofthese systems are template based. That is, templates designed by thegraphic designers are provided to the computer system. Users set theircontent in the placeholders defined in the template with someflexibility to change colours or fonts.

Some more sophisticated design software programs perform an automaticlayout method using constraint-based optimisation techniques. Theautomatic layout method starts with a layout template and content isfilled in the placeholders of the template. The layout is then refinedon the basis of satisfying every predetermined constraint. The focus ofthe automatic layout methods is to optimise the refinement method of theinitial page layout.

Other systems are available that are directed to generating an initialpage layout. Some of these systems allow users to select a template;others automatically suggest a template from user's requirements. Sincea layout template is authored without knowing the external environmentcontext of the new layout design, the resulted designs could beinadequate. For example, a composite image page's background of which isformed by a red colour may be inappropriate because the wall where theimage will be displayed is red. In this instance, the wall is consideredto define the external environment context of the composite image andits page layout. In some of the systems, after a layout has beengenerated, colours used in the layout can be adapted to some aspects ofthe environment context. However, these available systems areineffective and lack flexibility. One such automatic layout systemgenerates composite images depending on the display context, e.g.display device size. For example, the layout generated for an HTML pagethat is to be displayed on a computer monitor, is different from thelayout generated for displaying on a mobile phone screen. However,visual context (i.e., external visual features associated with the pagelayout to be designed) is not considered in creating the page layout.

SUMMARY

According to one aspect of the present invention, there is provided anelectronically implemented method for generating a layout for acomposite image, the method comprising the steps of;

receiving at least one content element to be included within thecomposite image;

receiving an external environment context image indicative of theintended external environment of the composite image, the externalenvironment context image being captured by an image capture device;

receiving data indicating at least one of a size of the composite imageand a location of the composite image within the external environmentcontext image;

generating layout styles for the at least one content element based onfeatures of the at least one content element, features of the externalenvironment context image and a provided location of the compositeimage; and

generating the layout for the composite image, the layout comprising theat least one content element, based on the generated layout styles.

According to a second aspect of the present invention, there is providedan electronic system for generation of a composite image, the systemcomprising;

communication means for receiving at least one content element;

communication means for receiving an external environment context imageindicative of the intended external environment for the composite image,the external environment context image being captured by an imagecapture device;

communication means for receiving data indicating at least one of a sizeof the composite image and a location of the composite image within theexternal environment context image;

processing means for generating layout styles for the at least onecontent element within the composite image, based on features of the atleast one content element, features of the external environment contextimage and a provided location of the composite image; and

processing means for generating the layout for the composite image, thelayout comprising the at least one content element based on the layoutstyles.

Another aspect provides a computer program product having recordedthereon a computer program for performing the method of the firstaspect.

Further aspects are also disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

At least one embodiment of the present invention will be described withreference to the drawings, in which:

FIG. 1 is a schematic diagram of a computer system on which the proposedmethod can be practiced.

FIG. 2 is a schematic flowchart showing a method of automatic pagelayout generation according to the present disclosure.

FIG. 3 is an example of a content table for a page layout.

FIG. 4 is an example of an external environment context image.

FIG. 5 shows two examples of relevant context images.

FIG. 6 is a schematic flowchart showing details of step 250 of themethod illustrated in FIG. 2.

FIG. 7A is a block diagram showing the input and output of an imagesegmentation unit.

FIG. 7B is a schematic flowchart of an example of a method of imagesegmentation.

FIG. 7C shows object regions of an image.

FIG. 8 is a region list table for regions returned from the imagesegmentation unit.

FIG. 9 is an example of segmented regions of an image.

FIG. 10 is an image region table listing regions extracted fromdifferent types of images.

FIG. 11 is an example of layout styles.

FIG. 12 is a flowchart illustrating the method of creating layoutstyles.

FIG. 13 is a schematic flowchart illustrating a method of selecting abackground image.

FIG. 14A is an example of layout panels.

FIG. 14B is a schematic flowchart for initialising minimal font sizesand minimum image dimensions.

FIG. 15A is a schematic flowchart illustrating a method of laying outcontent.

FIG. 15B is a flowchart for a method of adjusting layout styles.

FIG. 16A and FIG. 16B show two page layouts.

FIG. 17 is an example of a page layout as a candidate for one designconcept.

FIG. 18 shows a test of a page layout on the specified location of theexternal environment context image.

FIG. 19 is a page layout with an additional object.

FIG. 20 is an example of a page layout with a unique background shape.

FIG. 21A and FIG. 21B illustrate different page layout sets.

DETAILED DESCRIPTION INCLUDING BEST MODE

The present disclosure aims to provide a method of automaticallycreating page layouts for composite images based on the visual contextwithin which the images would be viewed. A page layout is the spatialorganisation of graphic objects. Examples of graphic objects include animage, line art, graph or text. Stylistic features of an image mayinclude colour and texture of the image. The text images are associatedwith styles that can include the colour and the font of the text. Thepage layout creation method involves creating graphic objects from rawcontent, such as text, and arranging the graphic objects based on theirsize, position and style.

The visual context is partially defined by an external environmentcontext image that shows the intended external environment of the image,which may include the physical location and background on which thecomposite image is to be displayed. The visual context of an image mayalso include a number of context images (hereinafter referred to asrelevant context images) that have some relevance to the content of thepage layout. The external environment context image is only a contextualreference and is not used directly in the page layout. However, one ormore of the relevant context images may be used either as a whole (forinstance as a background) or in part, in the composition of the pagelayout.

The method for creating layouts of composite images based on the visualcontext of the layout will now be disclosed with reference to thefigures listed above.

System Implementation

The discussed method for image layout generation can be electronicallyimplemented either by way of a computer system 100, shown in FIG. 1, orin a dedicated image generation device (not shown). The followingdescription discusses mainly the hardware associated with computerimplementation of the proposed method. However, it will be appreciatedthat a method implemented in a dedicated image processing device wouldbe conceptually similar in both its implementation steps and hardwarerequirements.

The steps of the discussed method may be implemented by way of one ormore application programs executable within either the computer system100 or a respective dedicated image processing device (not shown in FIG.1). In particular, the various steps of the method are effected bysoftware instructions carried out either within the computer system 100or in a respective image processing device. The instructions may beformed as one or more code modules, each for performing one or moreparticular tasks. The software may also be divided into two separateparts, in which a first part and the corresponding code modules performthe described method. A second part and the corresponding code modulesmanage a user interface between the first part and the user. Thesoftware may be stored in a computer readable medium, including thestorage devices described below.

The executable software is loaded into the computer system 100 from thecomputer readable medium, and then executed by the computer system 100.A computer readable medium having such executable software or computerprogram recorded on it is a computer program product. The computerreadable medium may, for example, include a compact disc (CD) or adigital video disc (DVD), which may be read on the optical disc drive112. The use of the computer program product in the computer system 100preferably effects the advantageous method for generating image layoutsbased on the visual context of the layout, the specific method beingdescribed hereinafter. The software may also be permanently residing onan internal hard drive 110 of the computer system 100 or an equivalentdedicated image processing device.

As seen in FIG. 1, the computer system 100 is formed by a computermodule 101, input devices such as a keyboard 102, a mouse pointer device103 and a camera 119, and output devices including a printer 115, adisplay device 114 and loudspeakers 117. An externalModulator-Demodulator (Modem) transceiver device 116 may be used by thecomputer module 101 for communicating to and from a communicationsnetwork 120 via a connection 121. The network 120 may be a wide-areanetwork (WAN), such as the Internet, or a private WAN. Where theconnection 121 is a telephone line, the modem 116 may be a traditional“dial-up” modem. Alternatively, where the connection 121 is a highcapacity (e.g., cable) connection, the modem 116 may be a broadbandmodem. A wireless modem may also be used for wireless connection to thenetwork 120.

The computer module 101 typically includes at least one processor unit105, and a memory unit 106, for example formed from semiconductor randomaccess memory (RAM) and read only memory (ROM). The module 101 alsoincludes a number of input/output (I/O) interfaces including anaudio-video interface 107 that couples to the video display 114 andloudspeakers 117, an I/O interface 113 for the keyboard 102, mouse 103and an optional joystick (not illustrated), as well as an interface 108for the external modem 116 and printer 115. In some implementations, themodem 116 may be incorporated within the computer module 101, forexample within the interface 108. The computer module 101 also has alocal network interface 111 which, via a connection 123, permitscoupling of the computer system 100 to a local computer network 122,known as a Local Area Network (LAN). As also illustrated, the localnetwork 122 may also couple to the wide network 120 via a connection124, which would typically include a so-called “firewall” device orsimilar functionality. The interface 111 may be formed by an Ethernet™circuit card, a wireless Bluetooth™ or an IEEE 802.11 wirelessarrangement.

The interfaces 108 and 113 may afford both serial and parallelconnectivity, the former typically being implemented according to theUniversal Serial Bus (USB) standards and having corresponding USBconnectors (not illustrated). One or more storage devices 109 areprovided and typically include a hard disk drive (HDD) 110. Otherdevices, such as a floppy disk drive and a magnetic tape drive (notillustrated), may also be used. An optical disk drive 112 is typicallyprovided to act as a non-volatile source of data. Portable memorydevices, such optical disks (e.g., CD-ROM, DVD), USB-RAM, and floppydisks for example may then be used as appropriate sources of data andinstructions to the system 100.

The components 105 to 113 of the computer module 101 typicallycommunicate via an interconnected bus 104 and in a manner which resultsin a conventional mode of operation of the computer system 100 known tothose in the relevant art. Examples of computers on which the describedarrangements can be practised include IBM-PC's and compatibles, SunSparcstations, Apple Mac™ or alike computer systems evolved therefrom.

Typically, the application programs implementing the discussed methodfor generating layouts, as well as any intermediate processing andcalculation results produced during the various steps of the method, areresident on the hard disk drive 110 and are read, processed andcontrolled and/or executed by the processor 105. Intermediate storage ofsuch programs and any data fetched from the networks 120 and 122 may beaccomplished using the semiconductor memory 106, possibly in concertwith the hard disk drive 110. In some instances, the applicationprograms may be supplied to the user encoded on one or more CD-ROM andread via the corresponding drive 112, or alternatively may be read bythe user from the networks 120 or 122. Still further, the software canalso be loaded into the computer system 100 from other computer readablemedia. Computer readable media refers to any storage medium thatparticipates in providing instructions and/or data to the computersystem 100 for execution and/or processing. Examples of such mediainclude floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM orintegrated circuit, a magneto-optical disk, or a computer readable cardsuch as a PCMCIA card and the like, whether or not such devices areinternal or external of the computer module 101. Examples of computerreadable transmission media that may also participate in the provisionof instructions and/or data include radio or infra-red transmissionchannels as well as a network connection to another computer ornetworked device, and the Internet or Intranets including e-mailtransmissions and information recorded on Websites and the like.

If necessary, application programs and the corresponding code modulesmay be executed to implement one or more graphical user interfaces(GUIs) to be rendered or otherwise represented upon the display 114.Through manipulation of the keyboard 102 and the mouse 103, a user ofthe computer system 100 and the application may manipulate the interfaceto provide controlling commands and/or input to the applicationsassociated with the GUI(s).

In the case of a dedicated image processing device, the proposed methodfor layouts generation may also be implemented by a dedicated hardwaremodule. Such a module may include digital signal processors, graphicprocessors or one or more microprocessors and associated memory modules.Furthermore, the module may include image acquisition device, such asthe digital camera 119. The image acquisition device may be used forobtaining an image of the external environment, which hereinafter willbe referred as external environment context image. This image may thenbe taken into account in the design of the generated image layout, aswill be described below.

Description of the Method

FIG. 2 illustrates a flowchart 200 for the method of creating a layoutof a composite image using the visual context for the image, wherein thevisual context includes the external environment context of the image.The method starts at step 210 where the processor 105 of the electronicsystem 100 for the generation of the composite image layout receivesfrom the user content images and/or text that will be included in thepage layout. At step 220, the processor 105 receives from the user anexternal environment context image, indicative of the intended externalenvironment on the background of which the designed layout is going tobe viewed. The external environment context image may be captured by thedigital camera 119 or another image acquisition device. At step 230, theprocessor 105 receives from the user data of the location of the pagelayout on the external environment context image and of an approximatesize of the page layout, e.g., 30 cm width and 40 cm height, via agraphic user interface, utilising display 114, keyboard 102 and mouse103. It is envisaged that, in case of a relatively uniform externalenvironment context image, the precise location of the page layoutwithin the external environment context image may not be consideredessential and the only provided data may be that of the size of the pagelayout.

At step 240, the processor 105 may receive from the user one or morerelevant context images which will be used to extract information toproduce the page layout. This step is optional, as indicated by thedotted outline. The context images may be stored on storage devices 109and 110 or external disc drives, such as optical disc drive 112.

In the present methods, a search engine or other method could be used toretrieve relevant context images from a database or semantic web withuser provided keywords. The data bases could be stored on LAN 122 or WAN120. In step 245, the processor 105 generates one or more page layoutsfor the given content based on features of at least one of the contentelements, features of the external environment context image and/or theprovided location of the composite image. Styles extracted from anyprovided relevant context images may also be used for this purpose. Instep 250, the processor 105 creates one or more page layouts for thegiven content based on the styles generated in step 145.

As more than one page layout can be produced at step 250, each pagelayout may be sent by processor 105 to a user for evaluation, at theoptional step 260. Step 260 usually includes the processor 105presenting each generated page layout on the background formed by theexternal environment context image and allowing the user to review theproposed layout. The step of presenting the image can be effected by theprocessor 105 displaying the proposed image layout on the display device114 or by printing the image on the printer 115. The best layout is thusselected by the candidate.

Content Elements

Each image or text specified in the content is called a content elementor element. Any non-text content element is defined as an image contentelement, e.g., a photographic image, line art, or clipart. Whileproviding the content in step 210, the user can specify the order andthe emphasis of the content elements in the final page layout.

The emphasis of the element indicates the level of importance that theuser wants to convey to the viewer of the page layout. The values ofemphasis considered here are A, B, C, and D, where A has the highestlevel of emphasis and D has the lowest emphasis. One or more contentelements can have the same emphasis. Whilst the order of the contentelements describes which content element is laid out first and whichcontent element is laid out last, the emphasis specifies to what degreeeach content element is highlighted and does not correspond to the orderof the elements.

An example of content elements to be included in an image layout isshown in a content table 300 in FIG. 3. There are three columns in thecontent table: content element 320, element type 330 and elementemphasis 340. There are four text content elements; 350, 370, 380 and390, and one image content element 360. Text content 370 is marked withemphasis A, which means that it needs to be the most eye-catchingelement. While the image content element 360 and the text contentelement 350 have emphasis B and C, respectively, text content elements380 and 390 have the lowest emphasis D. The order of the contentelements is the order of rows in the content table shown in FIG. 3.

The styles that affect the visual properties of the content are calledlayout styles. No matter where a content element is laid out, itsemphasis can be changed by layout styles.

The emphasis is an important factor for the layout method to determinewhat layout styles to use. Usually, a big font draws the viewer'sattention. However, the emphasis due to a style is not inherent, butdepends on the style itself, the background, the layout and so on. Forexample, even text with a big font can remain unnoticed if the textcolour has very little contrast with the text's background colour. Moredetailed discussion of this issue is provided further in the text.

External Environment Context Image

The external environment context image utilised at step 220, is animportant input for creating a page layout according to the describedmethod. To make a page layout stand out from a background colour orsurrounding colours, the page layout must avoid using those colours. Forexample, if an advertisement is to be placed in a big sign board infront of trees, the page layout of the advertisement must not have abackground with the tree colour, e.g., green colour. Thus, the externalenvironment context image provides restrictions on selecting the layoutstyles.

As indicated above, the external environment context image indicates theenvironmental background in which the page layout is going to bedisplayed and viewed. The external environment context image could be aphotograph of a wall, a noticeboard or a window of a shop. FIG. 4 showsan external environment context image in the form of a noticeboard 400.The noticeboard 400 has a background 410. Six stickers 431, 432, 433,434, 435 and 436 and an A4 page 440 with some text 450 are pinned to thenoticeboard. The box with dotted line 460 indicates both the location ofthe designed composite image layout and the size of the area of thelayout. The associated data is provided by the user to the electronicsystem 100 in step 230.

Relevant Context Image

The relevant context images that may be received by the processor 105from the user in step 240 are a kind of visual context that can bedirectly used or indirectly applied as layout styles to the content.FIG. 5 illustrates two relevant context images 500. Image 510 shows amoonlight scene which is related to the expression “Moonlight” from text370. Image 520 is another relevant context image that depicts a cello590 which is related to “String Concert” from the same text 370. In therelevant context image 510, the top part 570 depicts sky and the bottompart 580 depicts water. Label 550 indicates the moon, while the waterreflection of the moonlight is labelled as 560. In the relevant contextimage 520, label 590 is used to identify the boundary of the wholecello, while labels 530 and 540 mark parts of the cello, and label 525identifies the background of the image.

The one or more styles generated by the processor 105 for the pagelayout of the newly designed composite image that is to be located at460, may at least partially be based on features of the relevant contextimages. The style includes attributes such as colour and shape. Forexample, the font colour of a text content element of the designed imagecan be set to the colour of the cello 520. It is to be noted that,whilst the described method does not include the step of recognising thecello 520, its boundary and colours are considered during the step 245of creating layout styles.

Now steps 245 and 250 from FIG. 2 are described in detail with referenceto FIG. 6. The method of creating page layouts 600 in FIG. 6 starts witha decision step 605 in which the processor 105 verifies whether there isan image received in steps 210, 220 and 240. If there is an image, theprocessor continues to step 610. Otherwise, the execution continues withstep 620.

Step 610 extracts regions from the selected received image. After theregions are extracted, the processor 105 returns to step 605 to verifywhether there are further images to process. All selected images aremarked so that they will not be selected again.

Step 620 creates layout styles based on the image regions extracted instep 610 and, thus, effects step 245. Then, step 630 selects anappropriate image as the background image of the new layout. Step 640lays out the content of the layout.

After a page layout is created, in step 650 processor 105 checks forother potential background images which were not examined in step 630.If there are any images left to process, the processor 105 goes back to630. Otherwise, the method terminates at step 699. At this stage theentire image layout has been generated and step 250 of FIG. 2 has beeneffected.

Further details of step 610, 620, 630 and 640 will be described furtherin the text in relation to FIGS. 7B, 12, 13 and 15A, respectively.

Extracting Image Regions

The main task of extracting image regions at step 610 from an imageobtained in 210, 220 or 240 is to segment the image into a list ofregions associated with the image. Thus, a preferred implementationemploys image segmentation techniques to extract image regions.

FIG. 7A depicts a block diagram 7A00 showing schematically the input andoutput components of the image segmentation process performed by animage segmentation unit (ISU) 7A40. It would be appreciated that theused references are only notional and the processing within the ISU 7040is actually performed by processor 105. In particular, ISU 7A40 receivesone image from 7A10, 7A20 or 7A30, a set of thresholds 7A50, and outputsa list of regions 7A60. The ISU subdivides a pixel image into itsconstituent regions or objects. Most of the segmentation algorithms arebased on one of two basic properties of intensity values: eithersimilarity or discontinuity. In the first category of the algorithm, theapproach is based on partitioning an image into regions that are similaraccording to a set of predefined criteria. The approach in the secondcategory is to partition an image based on abrupt changes in intensitysuch as edges. Both techniques are used in one embodiment of thedescribed method.

FIG. 7B depicts a flowchart of an image segmentation method 7B00 in oneimplementation. The method starts with step 7B10 of quantising andsegmenting the image into regions with similar attributes. The image isfirst divided into 32 by 32 pixel tiles. Then each tile is examined todetermine a set of quantised colours that best represent the tile. Inthe preferred embodiment, the maximum number of quantised colours in asingle tile is 4. However, any suitable number of quantised colours maybe used. The pixels of the image are segmented, using the quantisedcolours, into a collection of regions. A region consists of neighbouringtiles that have similar attributes. Attributes are described bystatistics such as dominant colours, average colours, number of pixels,pixel density and ratio of perimeter to area of the pixels. The set ofthresholds 7A50 may contain a colour similarity threshold value ThC tobe compared against colour similarity values calculated as follows. Inone embodiment, ThC can be 5 pixels. If a pixel is represented by (Rp,Gp, Bp) which are red, green and blue values, and the average colour ofthe region is (Rm, Gm, Bm) for red, green and blue values, the pixel isconsidered as having a similar colour to the region under the followingconditions:

|Rp−Rm|<ThC AND |Gp−Gm|<ThC AND |Bp−Bm|<ThC

Step 7B20 converts the regions obtained from 7B10 into connectedcomponents (CCs) which consist of adjacent pixels found within theregions. In the preferred embodiment, a tree graph is generated based onthe containment of the connected components. A first CC is contained bya second CC if none of the first CC contributing pixels exist outside ofthe second CC. For example in FIG. 5, CC 560 is contained in CC 580 andCC 540 is contained in CC 530.

After producing the connected component tree graph, a process of textregion detection starts. At step 7B30 a CC is obtained. Next, decisionstep 7B35 checks whether the CC is a text CC using its statistics withpredefined font information. Examples of the statistics used hereinclude: size, density, aspect ratio, straightness and the relationshipbetween the current CC and any nearby CCs. The CC relationshipinformation may be distance between the edges of the CCs, differences inthe coordinates between the bounding boxes of the CCs or any othersuitable information.

If a CC is a text CC, the process continues to step 7B40 to add the CCto a text region adjacent to the CC. If no text region is nearby, a newtext region is created. A text region consists of one or more lines oftext. The distance threshold, predefined in 7A50, is between a CC and atext region. If the distance between a text CC and any text region isgreater than this distance threshold, e.g., 2 pixels, a new text regionis created and the CC is added to the new text region.

After step 7B40, the process continues to step 7B45 to find out if thereare more CCs in the CC tree graph. If there is another CC, the processgoes back to step 7B30. Otherwise, it continues to step 7B50.

If in decision step 7B35 a CC is not a text CC, the process goesdirectly to step 7B45.

When there are no more CCs to process, the process goes to step 7B50which finds close boundaries from pixels not belonging to any of thetext regions. In one embodiment, an edge detection method using Sobelmasks is used to find edges. Following the edge detection process, anedge linking technique is applied to find the region boundary. Oneapproach is regional processing that uses functional approximation tofit a 2-D curve to the known points. Polygonal approximations can beused to capture the essential shape features of a region.

Step 7B60 creates object regions from the boundaries obtained from step7B50 and references to the regions from 7B10 that are enclosed insideeach boundary. In other words, each object region is defined with aboundary and a list of regions. The statistics of the object regioncontains the statistics of the enclosed regions and statistics of theobject, e.g., the ratio of perimeter of the boundary of the object toits area. In FIG. 5, the relevant context image 520 has a backgroundregion 525 and an object region 590. The regions extracted from 7B10 forthe cello 590 have the statistics about the colours of different regionsof the cello, including 530 and 540.

Regions obtained from step 7B10 that are not classified as text regionsor included in any object regions are referred to as simple regions. Forexample, 410 in FIG. 4 is a simple region.

Step 7B70 groups all regions into three types: text regions, objectregions and simple regions. The method of image segmentation concludesat step 7B80. The output of the ISU 7A40 is a list of regions 7A60.

FIG. 7C shows labels indicating the object regions of a content image7C00. Objects 7C10, 7C20, 7C30, 7C40, 7C50, and 7C60 are detected asobject regions. 7C70 is one of the regions contained in the objectregion 7C50.

FIG. 8 illustrates an example of a region list with three region types.The region table has columns for region id 810, region type 820 andstatistics 840. A region type can be text, object or simple. However,other region types can also be used by the described method.

Each row contains statistics for the simple region, for the objectregion or for the text region. In the preferred embodiment, thestatistics can be:

-   -   colours (e.g., average colour, dominant colour, colour        distributions)    -   fonts if it is a text region (including size, font family,        weight, etc)    -   position    -   bounding box    -   boundary    -   ratio of boundary circumference to its area

The font statistics only appear in the text region 847, while othersapply to regions of all types, 845, 846 and 847.

There are region statistics in rows 850, 860 and 870 for three differenttypes of regions: simple region, object region and text region. Forexample, average colour (yellow) is defined in 845 as a statisticalcharacteristics of the simple region 850, boundary of the object isdefined in 846 as a statistical characteristics of the object region 860and font size and background colour (blue) are defined in 847 as astatistical characteristics of the text region 850.

FIG. 9 shows an example of a segmented image. The background of theimage 900 is a simple region labelled as 910. There are three objectregions; 920, 930, 940 and one text region 950. Depending on thethreshold for the image segmentation, the regions can vary a great deal.For example, if the colour similarity threshold in 7A50 increases, alarger region may be detected, e.g., regions 920 and 930 could be mergedinto one region, since more colours will be considered as similar.However, the text region 950 and the non-text region 920 cannot bemerged because their region types are different.

Creating Layout Styles

The statistics of the regions form the styles of the image. As theextracted styles from different image types serve different purposes forthe image layout creation process, they must be organised separately.FIG. 10 is an example of the image region table 1000 for images 400,510, 520 and 7C00. The table consists of columns: image id column 1001,image type column 1002 and region list column 1006. In this example,there is one row 1050 for content image 7C00, two rows 1064 and 1066 forrelevant context images 510 and 520 respectively, and one row 1070 forenvironment image 400. Only text regions and object regions are shown inthe region list 1006 in FIG. 10. For each of the content image, tworelevant images, and the environment image, associated object regions1055, 1065, 1067 and 1075 respectively are listed.

The statistics of the extracted regions from all three kinds of imagesare used to create styles to lay out the content of the newly designedcomposite image. The styles applied to the content for the page layoutare called layout styles.

There are different layout styles that can be applied to differentcontent elements to determine visual properties relevant to therespective layout element. For example, for laying out a text element,layout styles include font colour, size, family, weight, etc. When thetext is laid out on the page, it is formatted inside a boundary. Thisboundary area and its location are also considered as a layout style forthe text. For an image, its styles include location, dimension andborder. However, there is no limit on what can constitute layout stylesin the present method.

Step 620 in FIG. 6 is now described in more detail.

FIG. 12 illustrates a flowchart 1200 of the step 245 of generating thelayout styles. This method creates three types of layout styles:

-   -   base styles: can be used directly for a content element and are        initially extracted from the relevant context images;    -   auxiliary styles: should be considered if no appropriate base        style can be applied to a content element, and are initially        extracted from the content images;    -   avoided styles: cannot be used for any content element and are        extracted from the environment image.

An example of layout styles created from images 400, 510, 520 and 7C00is shown in FIG. 11. There are columns 1110, 1120, 1130 and 1140 forstyle type, colour palette, shapes and font, respectively. Three styletypes; base styles 1150, auxiliary styles 1160 and avoided styles 1170,are included in 1100. The colour palette 1120 has a list of colour 1188described in RGB and a percentage of usage 1190 in the original image.The colours are ordered by decreasing percentage used. The shape list1130 contains selected object regions from region list 1006. Shape 1158is an example of a shape belonging to one of the base styles 1150. Shape1180 is a rotated copy of shape 1158 and is for auxiliary styles 1160.The shapes listed in 1175 are for avoided styles 1170. The shapesdescribed by region boundaries are arranged in the order of area.However, they could also be ordered by likelihood of usage. The shapesare mainly used as text formatting boundaries. In FIG. 11, thestatistics of each style (colour, shape or font) can be obtained fromthe statistics column 840 in the region list in FIG. 8. The layoutstyles 1100 are constructed during the step 620 of creating layoutstyles.

The method 1200 of creating layout styles for the composite image isillustrated in FIG. 12. The method begins with step 1210 in which theprocessor 105 constructs three colour palettes for base styles 1150,auxiliary styles 1160 and avoided styles 1170. Colours in the basestyles 1150 are from relevant context images, colours for the auxiliarystyles are from content images, while colours for the avoided styles arefrom the external environment context image. For each type of image, thedominant colours are extracted from the statistics 840 of the regionlist 800. A limited number (e.g. 5) of dominant colours are added to thecorresponding colour palette. In FIG. 11, there are only three coloursin the base styles 1150, five colours for auxiliary styles 1160, andfour colours for avoided styles 1170. The colours in the palettes are inthe order of most dominant colour first. The dominant colour isrepresented by a percentage of colour use in the original image. Eachcolour 1188 in the colour palettes is associated with a percentage 1190of that dominant colour to the original image.

Next, in step 1212 the processor 105 copies shapes of all objects andall fonts from the external environment context image to avoided styles1170. As the extracted styles from the external environment contextimage are external to the page layout design, they are used differentlyfrom the extracted styles from the content and relevant context images.To make the page layout stand out from the environment context, the pagelayout should avoid using any colours, fonts, or shapes of objects fromthe external environment context image.

Next, in step 1215 the processor 105 modifies colours in the base stylesand auxiliary styles by removing any colour which is close to a colourin the avoided styles 1170 or duplicated between base styles andauxiliary styles. For example, after step 1210, there are colours (c1,c2, c3) in the base styles 1150, colours (c2, c4, c5) in the auxiliarystyles 1160, and colours (c1, c6, c7, c8) in the avoided styles 1170.Step 1215 will remove c1 from the base styles because c1 also appears inthe avoided styles. Also c2 in the auxiliary styles is eliminatedbecause c2 is already in the base styles. Now, the three colour palettesare; (c2, c3) for base styles, (c4, c5) for auxiliary styles and (c1,c6, c7, c8) for avoided styles.

In step 1220 the processor 105 selects an object region from the regionlist 1065 or 1067 in the corresponding relevant context image 1064 or1066. If there are more than one relevant context images, after allobjects have been selected from one relevant context image, an objectfrom another relevant context image is considered.

In decision step 1225, the processor 105 compares the shape of theselected object with object shapes in the environment image, describedin the region list 1075. If the shape of the selected object is similarto the shape of any object from the environment image, the method, asexecuted by the processor 105, goes to step 1235. Otherwise, step 1230adds the shape of the selected object to the shape list of the basestyle 1150. Then the method proceeds to step 1235.

In step 1235 the processor 105 checks whether there are any more objectsin the relevant context images. If there are, the processor 105 returnsthe method back to step 1220. Otherwise, the method continues to step1238.

Step 1238 checks whether there is any font found in any of the relevantcontext image. If so, the font is added to the base styles 1150 in step1240. Then, the method goes back to step 1238 to check for another font.If there are no more fonts to select, the method continues to step 1260.

At step 1260, the processor 105 can added to the colour palette in theauxiliary styles 1160 a limited number of colours suggested from acolour adviser or user-defined rules (e.g., complementary colours to thecolours obtained from the external environment context image). Shapesthat may not be appropriate as a text boundary may be deleted ortransformed. For example, a narrow shape (e.g., region's width issmaller than the width of the longest word in a text content) is notsuitable for horizontally formatted text or text with long words.However, if the shape is rotated 90 degrees, it can be considered. Forexample, in FIG. 11 a rotated cello object 1180 is added to theauxiliary styles.

If there are too many layout styles created in previous steps, furtherselection rules can be created to determine which styles are mostappropriate for the page layout. For example, in the externalenvironment context image, colours appearing within 20 cm of the pagelayout position, listed in the colour palette of the avoided styles1170, should not be selected. Another rule can be choosing a colour fromthe colour palette that has the most contrast to the background colour.

The method 1200 terminates at step 1290.

Selecting Background Image

In the disclosed method, a background image may be considered for thepage layout. Any content and relevant context image is a candidate.However, only an image that has enough empty regions to lay out thecontent is suitable for a background image. The regions extracted fromstep 610 can be used to aid in this selection process.

Now, step 630 of selecting a background image is described in detail.FIG. 13 is a flowchart for the method 1300 of selecting a backgroundimage. First, step 1305 initialises minimal font sizes, image widths andheights for four levels of emphasis of the content elements. This stepis explained further with regard to FIG. 14B.

Once the font sizes are initialised, the estimate of total content areais performed in step 1310. As each text content element has an emphasis,it is assigned an appropriate font size according to its emphasis. Aninitial total text area can be calculated using these initial font sizesand text lengths. Each content image also has a minimum dimension. Thetotal area of the image content and text content can be calculated.

All content images must be present in the page layout, but theirdimensions may vary depending on their levels of emphasis. An imagecontent element can be selected as a background that will overlap withother content elements, or be laid out as an individual content element.However, the relevant context images can be included in the page layoutonly as a background or just used to extract styles.

In the selection process of a background image, each image from thecontent and relevant context images is examined. An image that has beenchecked will be marked. In step 1320, the processor 105 checks and ifthere is any unmarked image, the method continues with step 1330.Otherwise, the method concludes at 1380, with no suitable backgroundimage selected. Note that the external environment context image is notconsidered in this process.

Step 1330 selects the first unmarked image and scales it to theapproximate dimension of the page layout keeping the original aspectratio. This scale factor is then used for all regions inside this image.

Step 1340 then finds empty regions (scaled) in the selected image, whichare suitable for the content. An empty region is a non text region thathas flat colours or is characterised by low contrast among the coloursinside the region. Other criteria can be created as a selection rule forselecting an empty region. For instance, if a region has a long narrowshape which may not be suitable for text formatting, then this regionmay be excluded from consideration for an empty region.

In the decision step 1350, the processor 105 checks whether there areenough regions for the content area calculated in step 1310. If theselected background image is a content image, the area for this imagemust be excluded from the content area. In step 1350, the followingcondition must be met for the content to fit into the empty region:

$\begin{matrix}{T_{ER} > T_{CA}} & (1) \\{{{Min}( {RA}_{i} )} \geq {{Min}( {CA}_{j} )}} & (2) \\{{{Max}( {RA}_{i} )} \geq {{Max}( {CA}_{j} )}} & (3)\end{matrix}$ where${T_{ER} = {{{total}\mspace{14mu} {empty}\mspace{14mu} {region}} = {\sum\limits_{i = 0}^{n}{RA}_{i}}}},{T_{CA} = {{{total}\mspace{14mu} {content}\mspace{14mu} {area}} = {\sum\limits_{j = 0}^{m}{CA}_{j}}}},{{RA}_{i} = {{Area}\mspace{14mu} {for}\mspace{14mu} {empty}\mspace{14mu} {region}\mspace{14mu} I}},{{CA}_{j\;} = {{Area}\mspace{14mu} {for}\mspace{14mu} {content}\mspace{14mu} {element}\mspace{14mu} j}},{n = {{total}\mspace{14mu} {number}\mspace{14mu} {of}\mspace{14mu} {empty}\mspace{14mu} {regions}}},\; {and}$

-   -   m=total number of content elements excluding selected content        image as the background.

If there are enough empty regions to fit all content, the selected imageis a candidate for the background of the page layout and the method 1300continues to step 1360. Otherwise, the method returns back to step 1320.Step 1360 assigns each of the empty regions as a layout panel and marksthe selected image as ‘used’.

Each layout panel has a bounding box. A big panel should be divided intosmaller panels according to predefined partition rules. For example, onerule is: if the ratio of panel width and the maximum font size selectedfor the content is greater than 80, than the panel is split into twopanels with equal width. The order of the layout panels is assignedaccording to certain rules. For example, one predefined order is fromleft to right and then from top to bottom based on the top left cornerof the bounding box of the layout panel.

FIG. 14A shows the layout panels 14A00 obtained from the relevantcontext image 510. The order of the layout panels is 14A10, 14A20, 14A30and 14A40. Other regions 14A50, 14A60 and 14A80 are not selected aslayout panels because 14A50 and 14A80 are not big enough to fit theminimum content element in 300 and 14A60 has no flat colour.

At the conclusion step 1380 of the method 1300, either an image has beenselected or no image has been found to be appropriate for use as thebackground. All images selected in step 1330 must be marked so that thesame image will not be examined again.

If there is more than one image that can be used as a background, thedisclosed method, executed by the processor 105, will create more thanone page layout. If no background image has been be selected, there willbe only one layout panel for the page layout and a default layout flowis set, e.g., a grid based layout flow. Alternatively, one of thethresholds 7A50 may be changed to merge small regions in the contentimages and relevant context images, after which the background selectionmethod 1300 is started again.

Now the step 1305 of the method 1300 is further described with referenceto FIG. 14B. This step calculates minimal font sizes of text contentelements for every level of emphasis, and minimum dimensions (width andheight) of image content elements for every emphasis level. As each textcontent element or image content element can be associated with anemphasis and there are four levels of emphasis in the presentimplementation, minimal font sizes for the text content in each emphasisneed to be initialised. Similarly, minimal widths and heights for theimage content in different emphases must be initialised.

The method 14B00 starts with step 14B10. The step assigns a minimal fontsize f₀ for the lowest emphasis D and a minimal increment of font sizeΔf. These two initial values f₀ and Δf depend on the size of the pagelayout. For example, if the page is A4 or A3 size, f₀ is set to 6 ptsand Δf is set to 2 pts.

The following step 14B30 initialises the minimal width w₀ and height h₀of an image for the lowest emphasis D and a minimal increment Δs. Givenan image of dimension 600×800 pixels, the image needs to be scaled to apoint that new width w and height h satisfy the following conditions:

w=w₀ AND h<h₀ OR w<w₀ AND h=h₀

The minimal width and height also depend on the page size. For example,if the page is A4 or A3 size, both w₀ and h₀ are assigned to 30 pixels.The minimal increment Δs is used to calculate how much to increase thewidth and height for the next emphasis level. For the example above, Δsis equal to 10 pixels.

Now step 14B40 initialises a variable i to 1 for emphasis D to calculateminimal sizes for emphasis C which is one level higher than the lowestemphasis D.

Decision step 14B45 then checks whether variable i is smaller than 4,which corresponds to the highest level of emphasis A. If i is less than4, the method continues to step 14B50. Otherwise, the method concludesat step 14B99.

Step 14B50 assigns the minimal font size, width and height for the nextemphasis level as follows:

f _(i+1) =f ₀ +i×Δf

w _(i+1) =w ₀ +i×Δs

h _(i+1) =h ₀ +i×Δs

Step 14B60 then increments variable i by one and returns back to step14B45 for the next level of emphasis. Once the highest level of emphasisis reached, the method 14B00 terminates in step 14B99.

Using the assigned text font sizes and image dimensions, the initialcontent area can be estimated at step 1310.

Laying Out Content

Now, step 640 of laying out the content of the composite image isdescribed in detail. FIG. 15A shows the method 15A00 of laying outcontent with the layout styles 1100 created in step 620. In oneembodiment, the method 15A00 makes even distribution of the contentacross the available layout panels created in step 630.

The method starts at step 15A10 by the processor 105 initialising layoutstyles for the content. Although the minimum font sizes of the textcontent and the minimum dimensions of the image content are initialisedat step 1305, they may not be appropriate for the available layoutpanels. For instance, they may be too small for the spaces in theavailable layout panels. Thus, in this step, font sizes of text contentelements and image dimensions are re-estimated. For example, all initialsizes are multiplied by the ratio of available layout panel space toinitial content area T_(ER)/T_(CA) (see (1)). Besides the font sizes,all text content elements are assigned font colours, font family etc.,in order to create graphic objects that can be laid out on the pagelayout. Initially, a content image which is already a graphic object hasno other layout styles, e.g., border, except its dimensions are adjustedusing the ratio T_(ER)/T_(CA).

Next, in step 15A20, the processor 105 initialises the layout panels.Each layout panel is associated with region statistics 840 described inFIG. 8. All layout panels are marked as available and no content elementis laid out on them.

In the decision step 15A25, the processor 105 checks whether there aremore panels to select for laying out content. If there is a layout panelavailable, the method continues to step 15A30. Otherwise, it goes tostep 15A75.

In step 15A30, the processor 105 selects an available layout panel fromthe ordered layout panels determined from step 1360. Once a layout panelis selected, a decision step 15A35, executed by the processor 105,examines whether any content element is selected for layout but has notbeen laid out yet. If no content element is selected, step 15A40 isexecuted by the processor 105 to select a content element to be laid outbased on the content order in 300. The content image used for backgroundis not considered. The method then continues to step 15A45. If in step15A35 the processor 105 finds that there is a selected element that hasnot yet been laid out, the method goes directly to step 15A45.

Step 15A45 is then executed by the processor 105 to check whether theselected layout panel has enough space to fit the selected contentelement. If it has, the method continues to step 15A50. Otherwise, themethod returns back to step 15A25 to select another available layoutpanel. All used layout panels must be marked as ‘used’ so that they willnot be selected again.

At step 15A50, before laying out a text content element, the font colouris checked with the layout panel's background. If the font colour is tooclose to the background colours extracted from the layout panel's regionlist 840, the font colour is reselected from the layout styles 1100. Thenew selected colour must have enough contrast with the backgroundcolour(s) and must not be among the colours defined in the avoidedstyles. A colour adviser can be used to calculate the font colour basedon background colours.

When the text content element has the highest emphasis and the area onwhich the text is to be laid out is smaller than any of the contentimages (provided that the content image is not a background), a shapecontour from the layout styles 1130 is considered to be applied for thetext formatting boundary to increase the visual emphasis of the text.The shape needs to be scaled to fit the selected text content elementand the text is formatted within the boundary of the shape. Then theshape with the text is placed in the layout panel according to thelayout flow.

More layout rules can be added to this layout method to change thestyles applied to the content element. For instance, a border can beapplied to a content image to raise the emphasis. The border width andborder colour can be calculated by the processor 105 based on thebackground colour around the image and the colours from neighbouringlaid-out content.

New statistics for the layout panels must be calculated based on thelaid-out content. For example, for a layout panel with text content, thefont colour and size, as well as the number of characters, contribute tothe colour distribution of the layout panel.

After laying out the selected content element in step 15A50, the contentelement must be marked as ‘used’ and the method 15A0 is executed by theprocessor 105 to continue to the decision step 15A55 that checks whetherthere are any more elements to be laid out. If so, the method isexecuted to return back to step 15A40 for selecting another contentelement. Otherwise, a page layout with laid-out content is created atstep 15A60 and then the method continues to step 15A75.

Decision step 15A75 includes automatic inspection of the overall look ofthe page layout. Similarly to most of the steps in the disclosed method,the processing associated with this automated inspection is conducted byprocessor 105 or by other processing means located on WAN 120 or LAN122. The inspection may, for example, include examination of all layoutpanels with laid-out content, the available space not yet used in thecurrent layout panel and the unused panels. If there is too much spaceleft, the method is executed by the processor 105 to continue to step15A85.

In step 15A75, other aesthetic criteria affecting the overall look canbe also checked. For instance, the emphases of the content elements mayalso be examined. If the original emphases of the content elements arenot kept, the method may also proceed to next step 15A85.

Other methods of checking the overall look can also be applied. Forexample, whether the overall look of a page layout is good or bad canalso be judged by verifying the compliance of the generated page layoutwith a series of predetermined aesthetic criteria including balance,content distribution, symmetry, emphasis, etc, usually defined fromdesign principles for graphic design. An aesthetic criteria of the pagelayout can be calculated based on region statistics. As the region'sstatistics contain local information about regions, the overallstatistics can be also built from the local statistics. If the overallcheck on the layout does not reach a desired limit, e.g., not balancedor symmetrical, the method can be executed to proceed, via step 15A85,to a step 15A90 of adjusting the layout styles.

Step 15A85 checks whether there are too many iterations for adjustmentof layout styles. If it is within a limited number of iteration (e.g.,10), the method is executed by the processor 105 to continue to step15A90 to adjust the layout styles and restart the content layout methodagain. If, at step 15A75 the generated page layout is found satisfactorythe method terminates at 15A99.

When there are no more layout panels found at step 15A25, the method isexecuted by the processor 105 to go to decision step 15A75 to verifywhether there are any more content elements to be laid out. If there arefurther elements to be laid out, the method continues to step 15A77,where verification is performed for whether the elements have beenassigned with minimum font sizes and image dimensions. If so, the methodterminates with an incomplete page layout which means not all contentcan be fitted into the available layout panels. However, if the contenthas greater sizes than the minimum, the sizes can be decreased, thus themethod proceeding to step 15A85.

The adjustment in step 15A90 includes adjusting the content sizes andemphases. In one example, the emphasis of a text content element may beassociated with implementing the aesthetic criteria with the followingstyles:

-   -   1. PT₁: Text area ratio to total content area;    -   2. PT₂: Contrast ratio (calculated from the font colour and        background colour, 0 if no contrast, 1 for maximum contrast);    -   3. PT₃: Text formatting boundary border factor (0 if there no        border, 1 if border width has the maximum pixels (e.g., 10        pixels);    -   4. PT₄: Text formatting boundary border contrast ratio        (calculated from boundary colour and background colour, value        ranges from 0 to 1);    -   5. PT₅: Text formatting boundary shape factor (e.g., 0 if is        rectangle, 0.5 if is circle, or 1 if shape is selected from the        layout styles. Other methods of calculating the shape factor can        be based on text area, perimeter, bounding box, etc).

If a style is described as PT_(i), the emphasis of a text element, Et,is described below:

$\begin{matrix}{{Et} = {\sum\limits_{i = 0}^{M}\frac{( {{PTi} \times {WTi}} )}{M}}} & (4) \\{M = {\sum\limits_{i = 9}^{M}{WTi}}} & (5)\end{matrix}$

where

-   -   WT_(i) is a predefined weight assigned with a value from 0 to 1;        and    -   M is the number of different styles listed above (e.g., 5) and        is the total of all weights, WT_(i).

In one example, the layout styles for image content element consist ofthe styles in the image itself and the border. The emphasis of image,Ei, can be calculated from the following styles:

1. PI₁: Ratio of image area to total content area;

2. PI₂: Contrast of dominant colour in the image and the backgroundcolour of the layout panel;

3. PI₃: Image border (0 if there no border, 1 if border has maximumwidth, e.g., 10 pixels);

4. PI₄: Image border contrast (calculated from border colour andbackground colour of the panel);

If each of the above styles is described as PI_(j), the emphasis of animage content element, Ei, is defined as below:

$\begin{matrix}{{Ei} = {\sum\limits_{j = 0}^{N}\frac{( {{PTj} \times {WTj}} )}{M}}} & (6) \\{N = {\sum\limits_{j = 0}^{N}{WTj}}} & (7)\end{matrix}$

Similar to (4), PI_(j) is from the list above and WI_(j) is the weightcorresponding to PI_(j). N is the number of different styles. In thiscase, N is 4. The summation of WI_(j) must equal N.

Each emphasis of a content element has a minimum value and a maximumvalue. The maximum value of the content element with the highestemphasis A is considered as the maximum emphasis value of the pagelayout and is denoted as E_(max). The minimum value of the contentelement with lowest emphasis D is considered as the minimum emphasisvalue of the page layout and is denoted as E_(min). The interval Ibetween E_(min) and E_(max) (E_(max)−E_(min)) is used to calculate thevalue of the emphasis in each level. In one embodiment, emphases aredefined as follows:

-   -   Emphasis A: I<Et≦0.8×I    -   Emphasis B: 0.8×I<Et≦0.6×I    -   Emphasis C: 0.6×I<Et≦0.4×I    -   Emphasis D: 0.4×I<Et≦0.1×I (emphasis below 0.1×I is not        acceptable)

Further, the position of the content element on the page can also affectthe emphasis. For example, an element position in the top left quadrantof the page has the emphasis calculated from formula (4) multiplied by afactor of 1.1, but an element in the bottom right quadrant has thecalculated emphasis multiplied by 0.9. This implies that even an elementwith less emphasis calculated from formula (4) ends up having moreemphasis because by default the top-left quadrant attracts moreattention. If a content image is selected as the background image, itsemphasis can be changed or ignored altogether.

At step 15A90, if the expected emphases of the content elements are notachieved after the layout process, any of the styles PT_(i) associatedwith the content element can be updated to change the emphasis. Forexample, increasing the contrast will increase the emphasis.

Two different page layouts 1600 and 1610 are shown in FIGS. 16A and 16B,respectively. Graphic objects 1650, 1660, 1670, 1680 and 1690 representcontent elements 350, 360, 370, 380 and 390, respectively. While thelayouts in these two figures are similar, there are obvious differencesin the object configuration and emphasis. For example, the differentcontrast used in FIG. 16B enhances the emphasis put on the text object1675, when compared with the object 1670 in FIG. 16A.

Now step 15A90 of adjusting the layout stiles is described in detail inrelation to one embodiment of the disclosed method, as shown in FIG.15B.

The method 15B00, effecting the step 15A90, starts at step 15B10, inwhich a ratio R is calculated by the processor 105 of the area of thelayout panels to the content area. The content area is the area occupiedduring the layout process. The processor 105 then compares, in step15B25, the ratio R with a threshold value Th defined in 7A50. Ideally,the content area should be evenly distributed in the layout panels andthe ratio R should be 1. However, an exact fit may not be possible forsome circumstances because of the text lengths and random shapes of thelayout panels. In this case, an extra space (e.g., 20%) in the layoutpanels, not occupied by any content, is allowed. For example, Th can beassigned a value of 1.2. If at step 15B25, R is found to be greater thanTh, there is too much empty region left. The method is executed by theprocessor 105 to go, via step 15B80, to step 15B32 to adjust the fontsizes and/or image sizes of content elements. Otherwise, the methodcontinues to step 15B30 of calculating the emphasis for each contentelement.

In step 15B32, processor 105 recalculates the font size for each textcontent element, and the width and height of each image in proportion tothe ratio R. However, the content area also needs to be recalculatedusing the new font sizes and image widths and heights. If the newcontent area is bigger than the layout panels, the sizes need to bedecreased. The calculation of the sizes can be iterative until thecontent area is smaller than the empty spaces of the layout panels andthe new ratio R is close to Th. If no new sizes can be obtained afterlimited iterations, as checked at step 15B80, the original sizes areused. That means the ratio R stays the same. If the number of iterationsto adjust font sizes and image dimensions 15B32 is larger than apredefined limited number, the method concludes at step 15B90. If thenumber of iterations to adjust font sizes and image dimensions 15B32 issmaller than a predefined limited number, the method goes to step 15B32.

Once the sizes of the text and images are fixed, step 15B30 calculatesthe emphasis of each content element using formula (4) or (6). Then step15B40 selects first content element and its emphasis. In the decisionstep 15B45, the processor 105 checks the current emphasis of theselected content element and its expected emphasis. If they aredifferent, the method is executed to continue to step 15B50. Otherwise,the method proceeds to step 15B55 to get next content element.

In step 15B50 the processor 105 updates the styles specified in formula(4) or (6) to obtain a new emphasis. For a text content element, if thecurrent emphasis is lower than the expected emphasis, some of thestyles, PT_(i) need to be increased. If the current emphasis is higherthan expected emphasis, some of the styles, PT_(i) needs to bedecreased. In the example shown in 1600, styles for 1670 with emphasis Aneeds to be changed to raise the level of emphasis higher than emphasisof 1660 (emphasis B). If the text area for 1670 is fixed, its text areapercentage to total content area PT₁ and the text formatting boundaryPT₅ are fixed. Thus, the emphasis of the text can be updated accordingto changes in the contrast PT₂, text formatting boundary border PT₃ andborder contrast PT₄.

First, contrast PT₂ is considered, i.e., increasing or decreasing thecontrast PT₂ of the selected content element according to its expectedemphasis. For example, graphic object 1670 for page layout 1600 ischanged to have higher contrast as shown in 1675 for a page layout 1610.Note that the font colour selection for calculating the new contrastmust check that the colours from avoided styles 1170 are not used. Thenew emphasis is calculated using the updated contrast.

If this new emphasis still does not meet the expected emphasis, the textformatting boundary border PT₃ is checked. If the emphasis needs to beincreased, a border width is to be increased. Initially, border width is0. Thus, no border is displayed. The border width has a maximum value ofe.g., 10 pixels.

If the new emphasis with the border cannot reach the required emphasis,text boundary border contrast PT₄ will be updated. Increasing the bordercontrast will also increase the emphasis.

If all the above adjustments fail to achieve the expected emphasis,changing the text formatting boundary using a different shape isconsidered. A shape from the base styles 1150 is selected and scaled tofit the text. Predefined selection rules can be added to select a shapethat is appropriate to the text length. Once the new text boundary isused, the style PT₅ is changed. The new emphasis is calculated.

Step 15B55 checks whether more content elements need their stylesupdated to meet their expected emphasis. If so, the method is executedby the processor 105 to proceed to step 15B60 to get the next contentelement and then the method returns back to step 15B45. Otherwise, themethod concludes at step 15B90.

FIG. 17 shows an example of a page layout created from step 250 usingthe relevant context image 510 as the background image and based onanother relevant context image 520 as the text formatting boundary fortext content 370.

The page layout 1700 can be placed in the external environment contextimage 400 as shown in FIG. 18. This test result 1800 can be displayed tothe user for evaluation, as described previously with regard to step260.

Variations and Extensions to the Method

The foregoing text describes only some embodiments of the disclosedmethod for automatic generation of layout of a composite image.Modifications and/or changes can be made to the disclosed method withoutdeparting from its scope and spirit, the discussed embodiments beingillustrative and not restrictive. Some examples of different specificimplementations of the method are discussed further in the text.

Additional Graphic Objects

In an alternative embodiment, if at step 15A75 a desired result cannotbe reached for the overall look of the page layout, additional graphicobjects may be laid out in a suitable position, e.g., empty space thatcould not be filled with any content. The additional graphic objects caninclude objects, such as line art and clipart, from the relevant contextimages or from a predefined library. It must have styles that can makethe page layout look balanced, or help the layout to achieve the overallaesthetic measure. For example, the space at the bottom right corner of1700 can be filled with a clipart object that will further improve thebalance of the proposed page layout. FIG. 19 shows another example of apage layout 1900, similar to 1700, but also including a cello 1930,extracted from the relevant context image 520.

Unique Background Shape

In an alternative embodiment, the step 630 can be updated to select abackground shape instead of a background image. The selected shape is aregion extracted from one of the relevant context images. In theparticular example shown in FIG. 20, page layout 2000 includes abackground in the shape of cello 2010.

Page Layout Set

In another embodiment, if the marked area 460 for the page layout in theexternal environment context image is too big (e.g., the ratio ofavailable layout panel space to initial content area (e.g.,T_(ER)/T_(CA), see (1) is greater than 100), the main page layout can besplit into multiple sub-page layouts. Step 640 can be updated to lay outcontent in multiple sub-pages.

Each sub-page layout, when considered individually, and all sub-pagelayouts, when considered as a set, must pass the predefined aestheticcriteria. This requires the inspection of the overall look at 15A75 tobe conducted both on each individual sub-page layout and on the combinedpage layout set.

The positioning of each sub-page layout is determined by predefinedrules. FIGS. 21A and 21B show two page layout sets 2110 and 2150,respectively. The page layout set 2110 has three rectangular sub-pages2120, 2130 and 2140 while the page layout set 2150 has four oval shapedsub-pages 2160, 2170, 2180 and 2190. The selection of shapes for thesub-pages and the background colour for the page layout sets 2110 and2150 must also follow the layout styles 1100.

External Environment Context Used for Different Purposes

In the above disclosure, the external environment context is used toextract avoided styles 1170 because the page layout is required to standout from the environment context. However, in different occasions theexternal environment context image can be used to create the base styles1150 and auxiliary styles 1160 for the page layout. For example, when apage layout is for an art poster to be placed in a living room, the artposter should be harmonious with the surrounding room in terms of thecolours and shapes of the objects in the room. In this case, the pagelayout can have layout styles including colours and shapes from theexternal environment context image.

Therefore, various rules on how to use the extracted styles from theexternal environment context image can be created to suit the purpose ofthe page layout design. The selection of the rules to be used in thepage layout generation method can be configured in step 230, after thelocation and area of the page layout is marked on the externalenvironment context image.

Layout Space Based on Background Image Emphasis

In some applications of the method, at least a portion of a contentimage may be utilised as a background image. The emphasis of thebackground image determines the amount of the layout space. Highemphasis of the background image implies that less space from thebackground image can be used to lay out the content elements. In otherwords, more background image contents are visible or not overlaid by thecontent elements. When the emphasis of the background image is low, morespace in the background image can be occupied by the overlaid contentelements.

An example of the relationship between the emphasis of the backgroundimage and the layout space allowed is shown below:

Background Image Emphasis Layout Space Allowed A 30% B 50% C 70% D 80%

Note that the allowed layout space T_(AS), is defined as the percentageof the total space of the background image.

During the execution of the layout method 640, the area of laid-outcontent must not exceed the layout space allowed. In step 15A10, thefont sizes of the text content elements and the image dimensions aremultiplied by the ratio of layout space allowed to the initial contentarea (T_(AS)/T_(CA)). In decision step 15A75, besides the overall lookbeing checked, the area of the laid-out content elements T_(CA), is alsocompared with the layout space allowed T_(AS). If there is too muchdiscrepancy between T_(CA) and T_(AS), the method proceeds to step15A90, via step 15A85, to adjust the font sizes and image dimensions.The discrepancy can be defined as a threshold in 7A50. The contentelements can be evenly distributed across the layout panels and thecontent elements laid out in each layout panel are centred.Alternatively, some layout panels may be fully overlaid with the contentelements while other layout panels may be left empty.

Changing Background Image to Reflect its Emphasis

When a content image is selected for a background image, its emphasisneeds to be reconsidered. For example, an object may be identified inthe background image called a background object (e.g., 550 and 560) thatis a content element with fixed position and size. The emphasis of thisimage can be either disregarded or changed. Usually the emphasis of animage, selected for background image, is reduced. Since the differentemphasis levels are effected by way of at least one of coloursaturation, brightness and contrast, to decrease the emphasis of abackground object, its appearance can be altered by methods such asdesaturation (i.e., removing the colours to grey scales), reducingbrightness, minimising its contrast or increasing its transparency.Other methods of reducing the emphasis of the background object can alsobe applied. This local modification of the background objects occurs instep 15A60, after all content elements have been laid out.

Alternatively, the entire background image can be updated to reduce theemphasis using the methods mentioned in relation to step 1330. Thus, thestatistics 840 of the regions in the background image are updated. Instep 1360, the assigned layout panels, 14A10, 14A20, 14A30 and 14A40,will have the modified statistics. During the execution of the method640 of laying out content, the new statistics for all layout panels areused to calculate the emphasis of each laid-out content element.

Dynamic Change of Page Layout

If the page layout is designed for an electronic display, the pagelayout may be arranged to update in response to changes in the externalenvironment context. For example, two photos can be taken of theexternal environment context of an electronic billboard at differenttimes. The first photo can represent the external environment contextduring the day, while the second photo can represent the externalenvironment context at night time. The photos can be received andprocessed by the disclosed method, either sequentially orsimultaneously. As a result, two page layouts are created, eachcorresponding to a respective one of the two external environmentcontext images. When the two page layouts are setup in the electronicdisplay, one is set to be displayed for the day time and another one isfor the night time.

An arrangement can also be made where photos can be taken periodically,or upon request. In this case the page layout can be dynamicallyadjusted in line with any changes in the respective external environmentcontext images indicated by a respective photo obtained subsequently tothe photo with the first external environment context image.

Interpretation of the Environment Visual Context in 3D

If the objects in the external environment context image can beidentified relative to the page layout location in 3D space, step 620 tocreate layout styles may be modified to consider the styles of theseobjects. For example, if the layout styles require styles from theexternal environment context image, the styles extracted from theobjects behind the page layout location may have higher priority thanthe styles from the objects in front of the page layout. Thus, thestyles with higher priority may be added to the base style 1150 and thestyles with lower priority may be added to the auxiliary styles 1160.

INDUSTRIAL APPLICATION

The described method is applicable to any industry related toadvertising, marketing and commerce.

1. An electronically implemented method for generating a layout for acomposite image, the method comprising the steps of; receiving at leastone content element to be included within the composite image; receivingan external environment context image indicative of the intendedexternal environment of the composite image, the external environmentcontext image being captured by an image capture device; receiving dataindicating a at least one of a size of the composite image and alocation of the composite image within the external environment contextimage; generating layout styles for the at least one content elementbased on features of the at least one content element, features of theexternal environment context image and a provided location of thecomposite image; and generating the layout for the composite image, thelayout comprising the at least one content element, based on thegenerated layout styles.
 2. A method according to claim 1, alsocomprising the step of: providing one or more relevant context images,each relevant context image being associated with one or more of the atleast one content element; wherein the step of generating layout stylesfor the at least one content element is also based on features of atleast one of the one or more relevant context images.
 3. A methodaccording to claim 2, wherein at least a portion of one of the one ormore relevant context images is included in the generated compositeimage.
 4. A method according to claim 1, wherein features of theexternal environment context image include at least one of: a colour ofthe external environment context image; an object shape in the externalenvironment context image; a background texture of the externalenvironment context image; and a font of a text region in the externalenvironment context image.
 5. A method according to claim 1, furthercomprising the step of: receiving additional information relating to anoccasion associated with the display of the composite image, whereingenerating the composite image is also based on the received additionalinformation.
 6. A method according to claim 2, wherein generating layoutstyles comprises the steps of: extracting one or more regions or objectsfrom image content elements, relevant context images and the externalenvironment context image; creating initial layout styles based onfeatures of an extracted region or object; adjusting the initial layoutstyles by comparing them with extracted styles from the externalenvironment context image; and accessing additional layout styles thatare in compliance with predefined selection criteria with respect tocolour, font or shape, for use as a text formatting boundary.
 7. Themethod according to claim 6, wherein; the step of extracting one or moreregions or objects from the at least one content element, the relevantcontext images and the external environment context image, is based onstatistics calculated for regions using image segmentation techniques,wherein the extracted regions are classified into text regions, objectregions and simple regions; and the initial layout styles are createdfrom the statistics calculated for the extracted regions or objects. 8.A method according to claim 1, wherein generating the layout of thecomposite image comprises the steps of: selecting from the relevantcontext images a background image to define a background within thecomposite image; and laying out the at least one content element on thebackground defined by the background image.
 9. A method according toclaim 8, wherein selecting a background image comprises the steps of:selecting an image from the at least one content elements and therelevant context images; extracting non-text regions from the selectedimage; if there is enough space in the non-text regions to fit the atlest one content element, choosing the selected image to be thebackground image. initialising non-text regions as layout panels andordering them in a layout flow.
 10. A method according to claim 9,wherein laying out content elements on the chosen background imagecomprises the steps of: initialising layout styles; initialising layoutpanels; laying out content elements in the layout panels according tothe layout flow to produce the composite image; inspecting the overalllook of the composite image; and if the composite image does not meet apredefined aesthetic criteria, adjusting styles to generate anothercomposite image.
 11. A method according to claim 10, wherein additionalgraphics objects are included in the background image for improving theoverall page layout of the composite image, the layout being evaluatedaccording to the predefined aesthetic criteria.
 12. A method accordingto claim 1, wherein generating the layout of the composite imagecomprises the steps of: selecting from the relevant context images abackground shape to define a background within the composite image; andlaying out the at least one content element on the background defined bythe background shape.
 13. A method according to claim 1, the methodcomprising creating a plurality of sub-page layouts which combine todefine the page layout of the entire composite image.
 14. A methodaccording to claim 1, wherein the layout styles for the at least onecontent element are generated so as to create contrast with the externalenvironment context image.
 15. A method according to claim 1, wherein atleast one content element is presented in the generated layout on thebasis of a respective emphasis level assigned to the element.
 16. Amethod according to claim 15, wherein the different emphasis levels areeffected by way of at least one of colour saturation, brightness andcontrast.
 17. A method according to claim 1, wherein a page layout iscreated with a background that comprises at least a portion of onecontent element.
 18. A method according to claim 17 wherein an emphasislevel associated with a content element, at least a portion of which isused as a background, is changed or disregarded.
 19. A method accordingto claim 1 wherein the page layout obtained on the basis on a firstexternal environment context image is dynamically adjusted on the basisof a subsequently obtained second external environment context image.20. A method according to claim 1 wherein a plurality of externalenvironment context images are simultaneously received and a pluralityof composite images is generated, each composite image corresponding toa respective external environment context image.
 21. A method accordingto claim 1 wherein, if objects in the external environment context imagecan be identified relative to the page layout location in 3D space, thestyles of objects located behind the page layout location are givenhigher priority compared to styles of objects located in front of thepage layout location.
 22. A method according to claim 1 wherein at leastone of the at least one content elements comprises text.
 23. A methodaccording to claim 1, the method further comprising at least one ofdisplaying and printing the generated composite image.
 24. A methodaccording to claim 1, wherein the step of generating the layout for thecomposite image further comprises an automatic inspection of the overalllook of the generated page layout.
 25. A computer program product havinga computer readable medium having computer program for electronicgeneration of a composite image, said program comprising executable codefor performing the method of claim
 1. 26. A computer program producthaving a computer readable medium having a computer program executableto generate a composite image, the program comprising: executable codefor receiving at least one content element; executable code forreceiving an external environment context image indicative of theintended external environment for the composite image, the externalenvironment context image being captured by an image capture device;executable code for receiving data indicating at least one of a size ofthe composite image and a location of the composite image within theexternal environment context image; executable code for generatinglayout styles for the at least one content element within the compositeimage, based on features of the at least one content element, featuresof the external environment context image and a provided location of thecomposite image; and executable code for generating the layout for thecomposite image, the layout comprising the at least one content elementbased on the layout styles.
 27. Electronic system for generation of acomposite image, the system comprising; communication means forreceiving at least one content element; communication means forreceiving an external environment context image indicative of theintended external environment for the composite image, the externalenvironment context image being captured by an image capture device;communication means for receiving data indicating at least one of a sizeof the composite image and a location of the composite image within theexternal environment context image; processing means for generatinglayout styles for the at least one content element within the compositeimage, based on features of the at least one content element, featuresof the external environment context image and a provided location of thecomposite image; and processing means for generating the layout for thecomposite image, the layout comprising the at least one content elementbased on the layout styles.